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PREFACE 


This  report  describes  activities  performed  during  the  development  of  interface  technologies 
designed  to  improve  supervisory  control  of  multiple  infonnation  sources  (711  HPW/RHCI), 
Work  Unit  71840919,  Supervisory  Control  Infonnation  Management  Research  (SCIMR).  The 
authors  thank  the  entire  SCIMR  team  to  include  Mr.  Antonio  Ayala,  Dr.  Guy  French,  Dr. 
Thomas  Carretta,  Dr.  Kristen  Liggett,  Mr.  James  Boyer,  Mrs.  Sarah  Lampke,  Mr.  Jimmy 
Whalen,  Mr.  Jason  Roll,  Mr.  Brian  Donnelly,  Ms.  Hannah  Combs,  and  Mr.  Sohom  Manna. 


v 


1.0  INTRODUCTION 


Since  its  inception,  the  Remotely  Piloted  Aircraft  (RPA)  has  adapted  to  military  life  and  has 
subsequently  become  an  integral  part  of  modem  day  warfare.  Although  unmanned,  this 
technology  remains  dependent  on  human  interaction  for  optimal  function.  The  Supervisory 
Control  Information  Management  Research  (SCIMR)  program  sought  to  explore,  develop,  and 
evaluate  novel  information  management  tools  such  as  controls,  displays,  and  decision  support 
aids  for  the  supervisory  control  of  multiple  RPAs.  The  multiple  research  efforts  conducted  to 
accomplish  this  goal  fall  into  four  categories;  Mission  and  Sensor  Management,  Information 
Scalability,  Advanced  Visualization,  and  Multi-Modal  Interfaces 

The  SCIMR  program  resulted  in  9  conference  papers  and  4  technical  reports. 


Explore ,  Develop,  and  Evaluate 
Novel  Information  Management 
Tools  for  Multi-UAS  Supervisory 
Control  &  Urban  Tele-Presence 


Information 

Scalability 


Figure  1.  Four  Research  Areas  of  SCIMR 
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2.0  SUMMARY  OF  COMPLETED  EXPERIMENTS 


2.1  Mission  and  Sensor  Management 


Mission  &  Sensor 
Management 


This  set  of  the  SCIMR  studies  developed  and  tested  advance  interface  concepts  to 
support  target  detection,  classification,  and  prioritization  for  operators  monitoring 
multiple  video  streams. 


2.1.1  Vigilant  spirit  control  station:  A  research  testbed  for  multi-UAS  supervisory  control 
interfaces 


Introduction 

Bridging  the  gap  between  rapidly  advancing  technology  and  the  human,  the  Vigilant  Spirit 
Control  Station  (VSCS)  serves  as  a  multi-faceted  facilitator  in  areas  ranging  from  research  to 
combat  missions.  The  result,  consequentially,  is  an  increase  in  the  efficiency  of  the  program  by 
enabling  a  single  operator  to  supervise  multiple  vehicles.  Streamlining  technology  is  tantamount 
to  the  program's  success.  Developed  with  this  in  mind,  VSCS  effectively  integrates 
sophisticated  advancements  for  the  purpose  of  strengthening  the  collaborative  relationship 
between  the  operator  and  the  UAV,  and  ultimately  serves  to  propel  this  multi-purpose  asset  into 
the  next  decade. 


Figure  2:  Screen  Shot  of  a  Typical  VSCS  Configuration. 


Publications 
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Rowe,  A.  J.,  &  Davis,  J.  E.  (2009).  Vigilant  spirit  control  station:  A  research  testbed  for  multi- 
UAS  supervisory  control  interfaces.  Proceedings  of  the  fifteenth  International  Symposium  on 
Aviation  Psychology,  Dayton,  OH,  pp.  287-292. 


2.1.2  Effects  of  Video  Display  Format,  Video  Frame  Rate,  and  Target  Alerting  on  Target 
Designation  Study 

Introduction 

The  use  of  small  remotely  piloted  aircraft  to  support  military  reconnaissance,  surveillance,  and 
target  acquisition  (RSTA)  operations  is  becoming  increasingly  widespread.  RSTA  tasks  require  a 
human  operator  to  continuously  monitor  infonnation  from  the  vehicles'  sensors  and  detect 
potential  targets.  This  is  especially  problematic  when  an  operator  must  divide  attention  between 
two  or  more  videos.  The  current  study  examined  the  effect  of  video  display  format,  frame  rate, 
and  target  alerting  on  several  objective  and  subjective  performance  measures. 

Methods 

The  study  had  2  parts.  Part  1  examined  the  effects  of  video  display  mode,  frame  rate,  and  target 
alerting  level  on  target  acquisition  perfonnance.  Part  2  examined  subjective  impressions  of  the 
image  quality/interpretability  of  the  different  display  formats. 

A  total  of  16  participants  used  a  version  of  the  Vigilant  Spirit  Control  Station  (VSCS),  set  up  to 
Monitor  two  UAV  sensor  feeds  under  the  experimental  conditions.  These  feeds  were  simulated 
using  a  commercial  off  the  shelf  product  from  MetaVR  called  the  Virtual  Reality  Scene 
Generator  (VRSG).  Both  task  performance  and  questionnaire  data  were  collected. 

The  Task  Perfonnance  Measures  of  Number  of  Hits,  Number  of  Misses,  and  Number  of  False 
Alanns  are  self-explanatory.  These  were  collected  for  all  study  conditions.  Three  additional 
measures  were  available  only  for  the  conditions  where  target  alerting  was  provided:  False  Cue 
Misses,  Correct  Rejections,  and  Incorrect  Rejections.  False  Cue  Misses  occurred  when  the  target 
alert  indicated  an  object  that  was  not  a  target  and  the  participant  did  not  detect  the  error.  Correct 
Rejections  occurred  when  the  target  alert  indicated  a  non- target  and  the  participant  correctly 
rejected  the  alert.  Incorrect  Rejections  occurred  when  the  target  alert  correctly  indicated  a  target 
and  the  participant  incorrectly  rejected  the  alert. 

Each  test  trial  lasted  approximately  10  minutes.  At  the  beginning  of  each  trial  the  experimenter 
was  responsible  for  commanding  the  simulated  aircraft  to  begin  road  surveillance.  When  the 
aircraft  arrived  at  the  starting  location,  the  experimenter  verbally  told  the  participant  that  the  trial 
has  started.  Participants  were  then  responsible  for  designating  targets  and  responding  to  alerts, 
depending  on  the  experimental  condition.  When  the  aircraft  arrived  at  the  end  location  the 
experimenter  verbally  notified  the  participant  that  the  trial  had  ended.  Subjective  workload  and 
usability  assessment  data  was  collected  by  questionnaires  following  each  trial.  After  completion 
of  all  trials,  participants  evaluated  the  image  quality/interpretability  for  the  different  presentation 
formats.  This  was  done  to  determine  subjective  preferences  for  the  formats  and  to  illuminate 
whether  perceptions  of  the  formats  were  related  to  objective  performance. 
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Conclusion 


The  goals  of  the  study  were  to  1)  examine  human-system  interface  (HSI)  display  factors  that 
may  affect  the  performance  of  UAV  MPOs  doing  a  RSTA  task  requiring  sustained  attention  and 
2)  develop  HSI  design  recommendations  to  enable  human  operators  to  remain  focused  and  alert 
in  order  to  mitigate  performance  decrements  often  observed  over  prolonged  periods.  Results 
indicated  that  for  the  objective  measures  of  target  acquisition  perfonnance  and  subjective 
workload,  there  was  no  clear  benefit  to  be  gained  from  a  mosaic  display  fonnat  or  a  high  frame 
rate.  A  reliable  target  alert  clearly  enhanced  target  acquisition  performance.  Problems  with  the 
way  the  mosaic  display  fonnat  was  implemented  may  have  contributed  to  its  unexpected  poor 
perfonnance.  Additional  studies  are  needed  to  examine  the  effectiveness  of  these  display  factors 
under  more  realistic  conditions.  These  studies  should  employ  a  longer  duration  task  to  examine 
perfonnance  decrements  over  prolonged  periods,  use  a  wider  range  of  target  types  (e.g.,  moving 
and  still  targets)  and  settings  (e.g.,  rural,  suburban,  and  urban),  and  additional  measures  of  image 
quality/interpretability  (e.g.,  Video  National  Imagery  Interpretability  Rating  Scales).  Other 
studies  should  be  performed  to  examine  the  utility  of  video  and  image  enhancement  techniques 
for  improving  the  interpretability  of  video  imagery.  Some  examples  are  algorithms  involving 
stabilization,  deblurring,  dehazing,  super-resolution,  and  object  tracking. 

Publications 

Carretta,  T.  R.,  Ayala,  A.,  French  G.  A.,  Gonzalez-Garcia,  A.,  Liggett,  K.  K.,  Rowe,  A.  J., 
Wright,  N.  F.,  &  Flach,  J.  K.  (2009).  Effects  of  video  display  format,  video  frame  rate,  and  target 
alerting  on  target  detection  performance  in  a  simulated  environment,  AFRL-RH-WP-TR-2010- 
0042.  Wright-Patterson  AFB,  OH:  Air  Force  Research  Laboratory,  Human  Effectiveness 
Directorate,  Crew  Systems  Interface  Division,  Supervisory  Control  Interfaces  Branch. 

Carretta,  T.  R.,  Rowe,  A.  J.,  &  Flach,  J.  K.  (2010).  Effects  of  Video  Display  Format,  Video 
Frame  Rate,  and  Target  Alerting  on  Target  Designation.  Proceedings  of  the  HCI-Aero 
conference,  Cape  Canaveral,  FL. 


2.1.3  Vigilance  Decrements  in  a  Sustained  Attention  Task:  Examination  of  a  Mitigation 
Strategy:  Study  1 

Introduction 

A  study  was  conducted  to  examine  the  effectiveness  of  perceptual  and  cognitive  intervention 
tasks  on  mitigating  vigilance  decrements  commonly  observed  in  sustained  attention  tasks. 
Sixteen  participants  were  randomly  assigned  to  one  of  two  experimental  intervention  conditions 
(perceptual  or  cognitive).  Half  of  the  participants  completed  a  45-minute  "No  Intervention" 
Control  trial  first,  followed  by  one  of  the  Intervention  trials,  also  45  minutes.  The  other  half 
completed  one  of  the  Intervention  trials  first,  followed  by  the  No  intervention  trial.  Following 
each  trial,  participants  completed  the  SSSQ  and  the  NASA  TLX. 

Methods 
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Several  types  of  data  were  collected.  These  included  objective  measures  of  perceptual  vigilance 
ability  and  target  acquisition  performance  (hits,  misses,  false  alarms),  demographic/background 
data,  and  subjective  measures  of  mood/stress  and  workload. 

The  study  began  with  a  pre-briefing  regarding  research  objectives,  procedures  and  informed 
consent.  This  was  followed  by  administration  of  the  Demographic  Data  Questionnaire.  Next, 
participants  completed  an  abbreviated  version  of  the  Dundee  Stress  State  Questionnaire  (DSSQ) 
called  the  Short  Stress  State  Questionnaire  (SSSQ)  to  establish  a  baseline  of  their  stress  level. 
Next,  the  12-minute  Perceptual  Vigilance  Task  (PVT)  was  administered  to  estimate  participants' 
perceptual  vigilance  ability.  Scores  from  the  baseline  PVT  were  correlated  with  objective 
measures  of  perfonnance  on  the  experimental  task  to  examine  relations  between  the  tasks. 
Following  the  short  vigilance  task,  participants  completed  the  SSSQ  to  assess  changes  from  their 
baseline  (pre-vigilance  task)  level.  They  also  completed  the  NASA  TLX  to  assess  subjective 
workload.  There  was  a  10-minute  break  following  completion  of  the  PVT  and  questionnaires. 

Next,  participants  were  randomly  assigned  to  one  of  the  two  experimental  intervention 
conditions  (perceptual  or  cognitive).  Half  of  them  completed  the  "No  Intervention"  trial  first, 
followed  by  one  of  the  Intervention  trials.  The  other  half  completed  one  of  the  Intervention  trials 
first,  followed  by  the  No  intervention  trial.  Following  each  target  acquisition  block,  participants 
completed  the  SSSQ  and  the  NASA  TLX. 

Those  doing  the  No  Intervention  trial  first  completed  a  3-minute  practice  session  during  which 
the  target  was  3  times  more  frequent  than  during  the  actual  experiment.  Task  perfonnance 
feedback  was  provided  (i.e.,  hits,  misses,  and  false  alarms)  during  the  practice  trial.  After  a  short 
break,  the  45 -minute  experimental  trial  followed  during  which  no  performance  feedback  was 
given.  Following  the  experimental  trial,  participants  completed  the  post-trial  SSSQ  and  NASA 
TLX  to  assess  their  subjective  stress  and  workload,  then  took  a  10-minute  break  before 
beginning  the  second  experimental  session.  The  procedures  were  similar  for  the  Intervention 
conditions.  For  these  conditions,  participants  began  with  a  3-minute  practice  session  during 
which  the  targets  occurred  3  times  more  frequently  than  in  the  experimental  session.  During  the 
practice  session,  participants  received  feedback  on  target  acquisition  perfonnance  (i.e.,  hits, 
misses,  and  false  alanns)  and  the  intervention  occurred  6  times.  After  a  short  break,  participants 
completed  a  45-minute  experimental  trial  during  which  no  performance  feedback  was  given.  The 
post-test  SSSQ  and  NASA  TLX  were  completed  following  the  experimental  session  to  assess 
participants'  subjective  stress  (mood  state)  and  workload. 

The  experimental  task  was  to  monitor  simulated  video  feeds  from  2  remote  fixed  sensors 
positioned  to  monitor  traffic  intersections  in  an  urban  setting  and  to  designate  targets/suspicious 
behaviors  as  they  were  detected.  Participants  were  instructed  to  designate  targets  and  respond  to 
interventions,  depending  on  the  experimental  condition.  Figures  X  illustrates  the  displayed 
imagery  as  viewed  by  study  participants.  Each  screen  (left  and  right)  displayed  information  for 
one  of  the  two  remote  sensors. 
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Figure  3:  Displayed  Imagery  as  Viewed  by  Study  Participants. 


Discussion 

As  expected,  a  general  decrease  in  objective  perfonnance  over  time  was  observed.  However, 
contrary  to  expectations  and  prior  research  (St.  John  &  Risser,  2007),  the  experimental 
intervention  tasks  did  not  reduce  decrements  in  target  acquisition  perfonnance  over  time,  nor  did 
they  reduce  subjective  workload.  Methodological  differences  between  the  St.  John  and  Risser 
study  and  the  current  study  that  may  have  contributed  to  differences  in  the  effectiveness  of  the 
vigilance  mitigation  interventions  in  the  two  studies  is  the  intervention  schedule.  St.  John  and 
Risser  implemented  their  interventions  with  simulated  psychophysiological  monitoring  where 
the  intervention  was  triggered  by  a  missed  target.  In  the  present  study,  a  simpler  approach  was 
taken  by  implementing  a  constrained  randomized  schedule.  Such  an  approach,  if  effective,  would 
eliminate  the  need  for  physiological  monitoring  to  achieve  perfonnance  benefits.  Alas,  the 
current  study  indicated  that  such  hopes  were  unjustified.  Whether  the  reason  for  different  results 
lies  in  the  differences  in  the  nature  of  the  experimental  tasks,  the  intervention  schedule,  or  some 
interaction  of  the  two,  it  appears  at  this  point  that  physiological  monitoring  may  be  necessary  to 
achieve  the  performance  benefits  of  a  system  based  mitigation  strategy. 

Publication: 

French,  G.  A.,  Carretta,  T.  R.,  Flach,  J.  K.  (2011).  Vigilance  Decrements  in  a  Sustained  Attention 
Task:  Examination  of  a  Mitigation  Strategy.  Proceedings  of  the  sixteenth  International 
Symposium  on  Aviation  Psychology,  Dayton,  OH. 

2.1.4  Mitigating  the  Effects  of  Vigilance  Decrements:  Study  2 

Introduction 

The  objective  of  this  study  is  to  examine  the  impact  of  techniques  designed  to  reduce  the  effects 
of  vigilance  decrements  typically  found  in  sustained  attention  tasks.  The  proposed  research  is  a 
follow-up  to  study  1 .  SCIMR  recently  completed  a  study  to  examine  the  effectiveness  of  a 
secondary  task  on  mitigating  vigilance  decrements  in  performance  typically  seen  in  tasks 
requiring  sustained  attention  over  long  time  intervals.  The  study  design  was  based  on  procedures 
employed  by  St.  John  and  Risser  (2007)  who  examined  the  utility  of  perceptual  and  cognitive 
interventions  for  mitigating  the  vigilance  decrement.  Study  2  will  examine  the  utility  of  a 
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variable  intervention  schedule  that  mimics  St  John  and  Risser's  (2009)  physiological-based 
pattern.  1  additional  study  is  planned.  This  study  will  replicate  St.  John  and  Risser  (2009)  with 
the  addition  of  a  no  intervention  control  group  to  provide  a  baseline.  If  results  of  that  study  are 
consistent  with  St.  John  and  Risser  (2009),  a  second  study  will  be  performed  using  a  more 
realistic  RSTA  task  where  a  single  operator  is  required  to  monitor  sensor  feeds  from  two  remote 
sources. 

Method 

Several  measures  were  collected  prior  to,  during,  and  following  the  experimental  target 
acquisition  task.  Objective  measures  of  performance  for  the  experimental  task  (ET)  included 
proportion  of  hits  and  correct  rejections,  and  number  of  false  alarms. 

During  an  Abbreviated  perceptual  vigilance  task  (AVT),  participants  monitored  the  presentation 
of  8-  by  6-mm  light  grey  capital  letters  that  consisted  of 'O',  'D',  and  a  backwards  'D'  centered 
on  a  video  display  screen.  The  letters  were  constructed  in  24-point  type  using  an  AvantGarde 
font  and  were  exposed  for  40  milliseconds  against  a  visual  mask  that  consisted  of  unfilled  circles 
on  a  white  background.  Participants  were  instructed  to  use  the  mouse  to  indicate  when  the  target 
letter  'O'  was  presented.  Responses  were  scored  as  hits,  correct  rejections,  and  false  alarms. 

NASA  Task  Load  Index  (NASA  TLX),  a  subjective  workload  assessment  measure  that  allows 
users  to  evaluate  their  interactions  with  human-machine  systems  was  collected.  A  Short  Stress 
State  Questionnaire  (SSSQ)  was  also  collected.  The  SSSQ  (Helton,  2004)  is  an  abbreviated 
version  of  the  Dundee  Stress  State  Questionnaire  (DSSQ).  It  consists  of  four  subscales:  mood 
state,  motivation,  thinking  style,  and  thinking  content.  Scores  for  three  factors  are  derived  from 
the  subscales:  task  engagement,  distress,  and  worry.  Additionally,  one  component  of  the  SSSQ 
assesses  participants'  perceptions  of  their  physical  and  mental  workload.  One  version  of  the 
SSSQ  was  administered  before  task  performance  and  the  other  was  administered  after  task 
perfonnance. 

The  study  began  with  a  pre-briefing  regarding  research  objectives  and  procedures,  informed 
consent,  and  collection  of  demographic  data  about  factors  that  may  be  related  to  performance  on 
the  target  detection  task  (e.g.,  age,  gender,  and  experience  with  similar  tasks).  Participants  then 
completed  the  SSSQ  to  provide  a  pre-study  baseline  of  their  stress  level.  Prior  to  examination  of 
the  effect  of  intervention  schedule,  a  baseline  was  established  for  participants'  perceptual 
vigilance  using  a  12  minute  abbreviated  vigilance  task  (AVT)  based  on  a  method  described  by 
Temple  et  al.  (2000).  These  baseline  data  were  used  in  the  data  analyses  as  a  covariate  to  control 
for  differences  in  perceptual  vigilance  among  the  study  participants,  hollowing  the  12  minute 
AVT,  participants  completed  the  SSSQ  to  assess  changes  from  their  baseline  (pre-study)  level. 
They  also  completed  the  NASA  TLX  to  assess  their  subjective  workloads. 

Participants  were  instructed  to  press  the  Space  bar  when  they  detected  a  target.  Leedback  was 
provided  on  hits,  misses,  and  false  alanns  during  the  practice  trial.  After  a  short  break,  the  45- 
minute  experimental  trial  followed  during  which  perfonnance  feedback  was  given  only  for  the 
intervention  task  .  hollowing  the  experimental  trial,  participants  completed  the  post-trial  SSSQ 
and  NASA  TLX  to  assess  their  subjective  stress  (mood  state)  and  workload. 
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The  experimental  task  was  identical  to  that  used  by  St.  John  and  Risser  (2007,  2009). 

Participants  monitored  simulated  snapshots  from  a  Remotely  Piloted  Aircraft  flying  along  a 
highway  and  were  instructed  to  designate  targets  as  they  were  detected.  The  event  exposure 
duration  was  400  ms  and  the  event  rate  was  1  per  2  seconds  with  a  critical  signal  (target)  rate  of 
three  per  minute.  The  critical  signal  was  a  truck  icon  that  was  slightly  larger  than  the  non-critical 
signal  (110  pixels  vs.  100  pixels).  All  signals  occurred  in  one  of  6  fixed  screen  locations. 
Participants  were  responsible  for  designating  targets  and  responding  to  interventions,  depending 
on  the  experimental  condition.  The  experimental  task  lasted  about  45  minutes. 

Discussion 

Contrary  to  St.  John  and  Risser  (2007),  inclusion  of  a  cognitive  intervention  did  not  improve 
perfonnance  beyond  that  observed  for  a  No  Intervention  Control.  Further,  contrary  to  St.  John 
and  Risser  (2009)  the  difference  in  perfonnance  for  the  Random  and  Physiologically-Based 
Intervention  schedules  was  not  statistically  significant.  St.  John  and  Risser  (2009)  reported  hit 
percentages  of  64%  and  70%  for  their  Random  and  Physiologically-Based  Intervention 
schedules  compared  with  61.4%  and  65.0%  in  the  current  study.  The  relatively  smaller 
improvement  in  the  current  study  for  the  Physiologically-Based  schedule  may  be  because  its 
implementation  was  not  triggered  by  a  missed  target  (St.  John  &  Risser,  2007)  or  by  indicators  of 
fatigue/inattention  (St.  John  &  Risser,  2009).  Even  so,  this  does  not  explain  why  perfonnance  for 
the  two  intervention  conditions  was  not  better  than  that  observed  for  the  No  Intervention  Control 
(62.5%)  condition  in  the  current  study.  It  is  unknown  whether  the  difference  in  results  for  St. 
John  and  Risser  (2007,  2009)  and  French  et  al.  (2011)  and  the  current  study  lie  in  the  trigger 
mechanism  for  the  cognitive  intervention.  Implementation  of  a  vigilance  decrement  mitigation 
intervention  in  an  operational  setting  would  be  greatly  facilitated  if  it  were  not  required  to  link  it 
with  physiological  indicators  of  inattention  in  order  to  achieve  effectiveness.  Regardless,  the  low 
correlations  between  scores  on  the  abbreviated  vigilance  task  and  the  experimental  task  suggest 
that  they  are  not  measuring  the  same  constructs,  (i.e.,  do  not  share  construct  validity).  Further, 
the  failure  to  replicate  previous  findings  casts  doubts  on  the  robustness  of  the  effectiveness  of  a 
simple  cognitive  intervention  task  for  mitigating  vigilance  decrements  in  performance  on  real- 
world  tasks  that  require  sustained  attention. 

Publication 

French,  G.  A.,  Carretta,  T.  R.  (2012).  Combating  Vigilance  Decrements  in  a  Sustained  Attention 
Task:  Examination  of  Two  Cognitive  Intervention  Schedules  for  a  Secondary  Task.  Paper 
presented  at  Human  Factors  and  Ergonomics  Society  Conference,  Boston,  MA. 

Carretta,  T.R.,  French  G.  A.  (2012).  Combating  Vigilance  Decrements  in  a  Sustained  Attention 
Task:  Examination  of  Two  Cognitive  Intervention  Schedules  for  a  Secondary  Task,  AFRL-RH- 
WP-TR-2012-0172.  Wright-Patterson  AFB,  OH:  Air  Force  Research  Laboratory,  Human 
Effectiveness  Directorate,  Crew  Systems  Interface  Division,  Supervisory  Control  Interfaces 
Branch. 
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2.2  Information  Scalability 


This  set  of  SCIMR  studies  focused  on  seamlessly  compressing  critical  net-centric 
information  onto  man-portable  interfaces 


2.2.1  Pilot  Study:  The  Potential  Interface  Issues  Associated  with  Scaled  Devices  for  the 

Control  and  Sensor  Exploitation  of  Multiple  Small  Remotely  Piloted  Aircraft  (RPA) 

Introduction 

As  remotely  piloted  aircraft  (RPAs)  become  more  prevalent  in  the  military  theater  and  as  these 
devices  diminish  in  size  and  cost,  the  demand  for  control  and  sensor  information  to  be  presented 
on  small  man  portable  devices  will  increase.  An  RPA  simulation  environment  was  used  to 
present  participants  with  three  devices  of  different  scale,  to  elicit  subjective  feedback  about 
interface  concerns  during  a  representative  reconnaissance,  surveillance,  and  target  acquisition 
(RSTA)  classification  task.  Participants  performed  the  RSTA  task  with  each  of  the  scaled 
devices  using  two  different,  notional  classification  editor  graphical  user  interfaces  (GUIs). 
Participants  were  encouraged  to  talk  aloud  to  identify  potential  problems  and  possible  fixes  to 
the  scaled  interfaces.  Participants  also  completed  a  usability  questionnaire  and  a  workload 
assessment  fonn  after  experiencing  each  device. 

Methods 

Participants  were  instructed  to  use  2  simulated  RPAs  to  provide  route  surveillance.  Participants 
were  told  which  preloaded  path  to  survey  and  then  activate  the  Dynamic  Path  Surveillance 
System  (DPSS).  The  DPSS  automated  vehicle  and  sensor  control  to  maintain  surveillance  of  the 
designated  path.  Participants  were  to  monitor  the  sensor  video  feeds  to  identify  objects  of 
interest  along  the  route.  Objects  could  be  either  stationary  or  moving.  Upon  detection  of  an 
object  of  interest,  participants  were  to  bookmark  the  video  and  use  one  of  two  classification 
editors  to  classify,  annotate,  and  assign  a  level  of  urgency  for  the  selection.  The  editors  differed 
in  several  ways  including  location  on  screen  and  information  presentation.  The  two  different 
classification  editors  and  how  they  appeared  on  screen  are  shown  in  Figure  X. 
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Figure  4:  The  Classification  Editors 

This  task  was  repeated  for  each  of  the  three  interfaces.  Participants  were  encouraged  to  talk 
aloud  about  likes,  dislikes,  and  interface  design  issues.  After  each  trial,  the  participants 
completed  a  questionnaire  to  rate  the  satisfaction  and  functionality  of  the  interface  as  well  as 
identifying  the  top  problems  and  respective  solutions. 

Discussion 

This  pilot  study  highlighted  the  need  for  future  research  into  interfaces  that  will  be  used  on  small 
scale  devices.  Although  the  VSCS  version  used  was  not  optimized  for  use  on  small  devices, 
most  participants  thought  that  performing  a  RSTA  mission  with  one  or  both  of  the  scaled  devices 
was  feasible.  Future  studies  will  use  some  of  the  lessons  learned  from  this  study  to  help  guide 
the  design  of  control  stations  that  will  be  run  on  smaller  scaled  devices. 

Publication 

None. 

2.3  Advanced  Visualization 


This  set  of  SCIMR  studies  investigated  the  utility  of  virtual  reality  technologies  to 
support  operator  supervision  of  multiple  vehicles  while  maximizing  performance 
and  decreasing  vigilance  decrements 
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2.3.1  Visual  Awareness  with  Nemaline  Target  Accessibility  Graphics  Evaluation 
(VANTAGE)  Studies 

Introduction 

As  the  capability  to  capture  video  and  still  imagery  through  sensors  on  remotely  piloted  aircraft 
(RPA)  and  fixed  sensors  continues  to  increase,  the  demands  on  the  decision  making  abilities  of 
the  operators  will  also  increase.  While  automation  may  eventually  take  some  of  the  sensor 
monitoring  load  off  of  the  operator,  until  then  the  burden  of  maintaining  continuous  over  watch 
of  a  target  will  fall  on  the  human  operators.  The  VANTAGE  studies  are  a  series  of  5  studies,  all 
of  which  used  the  same  basic  task.  The  task  required  the  participants  to  maintain  a  clear  line  of 
sight  to  a  designated  target.  The  target  was  walking  around  a  generic  Afghanistan  urban 
environment.  The  participants  had  2,  4,  or  6  (depending  on  the  experiment  and  treatment 
condition)  remote  piloted  aircraft  to  keep  track  of  the  target  with.  The  vehicles  were  flying  in 
fixed  orbits  at  about  1000  feet  above  ground  level  (AGL),  which  created  obscurations.  To  help 
mitigate  the  obscurations  created  by  the  buildings  augmented  reality  was  used  to  show  the  user 
the  target  sensor  relationship,  specifically  the  rays  shown  in  figure  5. 


Figure  5:  Synthetic  Environment  with  Augmented  Reality  Overlay  (Rays). 

2.3.2  Predictive  Continuous  Vantage  Study  1 

Experiment  Particulars 

The  participants  were  trying  to  maintain  persistent  stare  on  a  stationary  target  (phase  I)  then  on 
moving  target  (phase  II)  under  varying  ray  conditions.  The  targets  for  both  phases  were  in  an 
urban  environment,  this  caused  the  moving  RPAs  to  at  times  lose  line  of  sight  to  the  target.  The 
varied  ray  attributes  were;  ray  color  (on/off),  ray  mouse  selectable  (on/off),  ray  provides 
obstruction  information  (on/off)  and  ray  thickness  meaning  (none,  RPA  distance  from  stare 
point,  resolution  of  the  sensor).  The  variations  made  twenty- four  unique  ray  conditions.  A 
baseline  condition  (no  rays)  was  also  investigated. 
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Discussion 


Overall  the  best  ray  configuration  was  selectable  colored  rays.  The  combination  led  to  an  11% 
increase  in  target  ID  accuracy  during  and  the  stationary  phase  and  3%  increase  in  the  percent  of 
time  the  target  was  unoccluded.  The  number  of  good  transitions  (transitions  to  a  sensor  that  had 
a  clear  line  of  sight  to  the  target)  was  also  greater  in  the  selectable  colored  condition,  however 
the  total  number  of  transitions  also  increased  in  this  condition.  Subjectively  the  moving  target 
portion  of  the  study  was  felt  to  be  more  difficult  than  the  stationary  target  phase.  The  different 
mission  phases  will  be  more  closely  examined  in  future  studies. 

Publications 

Rowe,  A.,  Venero,  P.,  &  Boyer,  J.  (2012,  June)  Improving  Situation  Awareness  Through 
Enhanced  Knowledge  of  Sensor  Target  Relationships.  Paper  presented  at  Military  Operations 
Society  Symposium,  Colorado  Springs,  CO. 

2.3.3  Predictive  Continuous  Vantage  Study  2 A 

Experiment  Particulars 

This  study  builds  on  the  insight  gained  from  'Predictive  Continuous  Vantage  Study  1',  starting 
with  target  behavior.  There  will  be  no  stationary  targets  in  this  study.  In  study  1  the  sensor  auto 
slewed  to  the  target  location  at  all  times,  in  this  study  the  participant  had  to  manually  slew  the 
sensor.  The  number  of  ray  conditions  was  down  selected  to  three.  The  three  ray  conditions  were 
none,  solid,  and  dashed.  Under  the  solid  condition  the  rays  were  only  colored  and  selectable. 
Under  the  dashed  condition  the  rays  were  colored  and  selectable,  but  also  provided  obstruction 
infonnation  by  going  dashed  when  the  line  of  sight  was  blocked.  The  other  independent  variable 
was  Vigilant  Spirit  Control  Station  (VSCS)  configuration,  master  screen  only  or  master  screen 
plus  four  individual  screens  for  the  sensors.  There  were  some  similarities  between  the  two 
studies  as  well.  In  both  studies  the  target  is  moving  through  an  urban  environment,  and  the 
participant  has  to  choose  one  of  four  RPA  sensor  feeds  to  provide  to  an  external  customer. 

Discussion 

Preliminary  results  showed  that  the  VSCS  configuration  independent  variable  had  no  effect  on 
performance,  however  the  ray  configuration  and  target  route  did.  The  solid  condition  improves 
percent  unoccluded  by  7.8%  and  3.8%  when  compared  to  the  no  ray  condition  and  the  dashed 
condition,  respectively.  The  route  that  the  target  followed  through  the  city  also  had  an  effect  on 
the  percent  unoccluded.  When  looking  at  the  unoccluded  time  span,  again  the  VSCS 
configuration  had  no  effect  but  the  ray  condition  and  target  route  did.  The  solid  ray  condition 
reduce  unoccluded  span  by  2.2  seconds  compared  to  the  no  ray  condition.  The  dashed  ray 
condition  reduce  unoccluded  span  by  1.7  seconds  compared  to  the  no  ray  condition.  There  was 
no  difference  between  the  solid  and  dashed  conditions. 

Publications 

Venero,  P.,  Rowe,  A.,  and  Boyer,  J.  (2012)  Using  Augmented  Reality  to  Help  Maintain 
Persistent  Stare  of  a  Moving  Target  in  an  Urban  Environment.  Paper  presented  at  Human 
Factors  and  Ergonomics  Society  Conference,  Boston,  MA. 
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2.3.4  Predictive  Continuous  Vantage  Study  2B 

Experiment  Particulars 

Similar  to  what  we  had  done  in  previous  studies,  we  told  the  participants  to  provide  the  best 
possible  sensor  feed  to  a  notional  joint  terminal  area  controller  (JTAC)  who  was  equipped  with  a 
small  hand-held  computer  (similar  to  an  Apple  iPad)  that  can  only  consume  one  video  at  a  time. 
In  this  scenario,  the  JTAC  was  relying  on  the  participant  to  maintain  a  constant  view  of  a  high 
value  individual  (HVI).  The  HVI  moved  through  an  urban  environment  for  approximately  eight 
minutes,  stopping  twice  per  trial  to  converse  with  associates.  The  trial  ended  when  the  HVI 
entered  a  building.  We  randomly  placed  six  RPAs  in  one  of  six  predetermined  starting  positions, 
and  they  flew  a  predetermined  route.  Due  the  path  of  the  target  and  the  route  of  the  RPAs,  no  one 
RPA  had  a  clear  line  of  sight  to  the  target  at  all  times.  We  gave  the  participants  an  introduction 
to  the  program,  and  they  reviewed  and  signed  the  infonned  consent  form.  They  then  completed 
three  training  trials  that  we  had  designed  to  familiarize  them  with  the  task  and  the  different 
aspects  of  the  control  station.  After  they  had  completed  the  training  trials,  they  completed  12 
data  collection  trials.  After  each  trial,  we  administered  questionnaires  that  assessed  workload  and 
situation  awareness.  The  experiment  had  two  independent  variables:  obstruction  information 
presentation  (OIP)  and  interface  configuration.  The  OIP  independent  variable  describes  how  we 
presented  the  obstruction  information  to  the  participant.  The  OIP  independent  variable  had  three 
levels:  none  (no  obstruction  information  presented),  timeline  only  (obstruction  information 
presented  through  the  timeline  only),  and  timeline  with  rays  (obstruction  information  presented 
through  timeline  and  the  vantage  rays).  The  interface  configuration  independent  variable 
determined  whether  or  not  the  operator  had  dedicated  screens  for  each  of  the  sensors.  Figure  3 
shows  an  example  of  the  map  (A),  master  sensor  screen  (B),  timeline  (C),  and  the  six  dedicated 
sensor  screens  (D). 


Figure  6:  Sample  Interface  with;  (A)  Map  (B)  Master  Sensor  Screen,  (C)  Timeline,  and  (D) 

Six  Dedicated  Sensor  Screens. 


We  collected  objective  and  subjective  measures  to  assess  the  participants'  performance.  We 
collected  the  following  objective  measures:  percent  unoccluded,  average  unoccluded  time  span, 
total  number  of  transitions,  and  number  of  good  transitions.  The  percent  unoccluded  was  the 
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amount  of  time  the  HVI  was  actually  visible  in  the  master  sensor  screen  divided  by  the  amount 
of  time  the  HVI  was  potentially  able  to  be  seen  in  the  master  sensor  screen.  The  average 
unoccluded  time  span  was  the  average  length  of  time  the  HVI  was  in  the  master  sensor  screen. 
The  total  number  of  transitions  was  a  count  of  the  number  of  times  the  participant  switched 
sensors  in  the  master  sensor  screen.  The  number  of  good  transitions  was  a  count  of  the  number  of 
times  the  participant  switched  sensors  and  was  able  to  see  the  HVI  in  the  new  sensor.  We  also 
collected  subjective  measures  for  the  operators'  situation  awareness  and  workload. 

Discussion 

The  underlying  belief  behind  the  hypotheses  was  that  the  obstruction  information  would  be 
beneficial;  however,  the  data  does  not  support  that  theory.  When  considering  the  performance 
metric  user  percent  unoccluded,  the  main  factor  that  improved  operators'  performance  was  the 
dedicated  sensor  screens.  This  data  does  not  support  hypothesis  hi  or  h3  (obstruction  information 
would  improve  operators'  performance).  Hypothesis  h2  postulated  that  the  timeline  would 
remove  or  reduce  the  need  for  the  dedicated  screen,  but  we  did  not  see  that  at  all.  When  looking 
at  the  transition  counts,  which  may  be  an  indication  of  the  operators'  workload,  we  see  no  benefit 
of  providing  the  participants  with  the  obstruction  infonnation;  in  fact  we  see  an  opposite  effect. 
The  question  is  'Why  did  we  see  this?' 

We  think  that  the  main  reason  providing  obstruction  information  did  not  help  operators  is  that 
the  database  for  our  environment  contained  noisy  information.  In  order  to  detennine  if  a  sensor  is 
being  blocked  or  not,  the  operator  needs  a  detailed  database  of  the  environment.  In  our 
experiments  we  had  such  a  database;  however,  it  did  not  account  for  architectural  details  (e.g., 
awnings)  or  other  attributes  in  the  environment  such  as  trees  or  vehicles.  These  minor  omissions 
from  the  database  would  cause  the  obstruction  information  driving  the  timeline  to  be  incorrect, 
which  would  in  turn  lead  the  participants  to  not  trust  the  timeline.  In  conclusion,  the  data  from 
previous  and  current  research  suggests  augmented  reality  rays  improve  operators'  performance 
when  maintaining  persistent  stare  with  multiple  sensors.  One  caveat  to  this  research  is  that  all  of 
the  tasks  took  place  in  an  urban  environment  with  straight  and  either  parallel  or  perpendicular 
streets,  possibly  helping  the  operators  in  unforeseen  ways.  The  number  of  RPAs  an  operator  is 
controlling  also  influences  the  utility  of  augmented  reality.  In  addition,  the  data  suggests  that 
providing  obstruction  information  is  not  beneficial.  Future  researchers  will  examine  the 
appropriate  size  for  the  dedicated  sensor  screens  and  begin  to  look  at  how  to  employ  this 
technology  on  mobile  devices. 

Publications 

Venero,  P,  Rowe,  A.  &  Boyer,  J.  (2013)  Information  Management  for  Multiple  Entities  in  a 
Remote  Sensor  Environment.  Paper  presented  at  Human  Computer  Interaction  International,  Las 
Vegas,  NV. 

2.3.5  Mobile  Vantage 

Experiment  Particulars 

This  study  builds  upon  the  work  of  Venero,  et  al.  to  detennine  the  optimum  number  of  RPA 
sensors  (2,  4  or  6)  an  operator  can  monitor  with  and  without  rays.  While  the  previous  study  used 
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a  desktop  monitor  and  mouse,  this  study  will  be  conducted  on  the  mobile,  touch-operated 
Samsung  Slate. 

Discussion 

The  hypothesis  for  this  experiment  was  that  the  ray  conditions  will  outperform  the  non  - 
ray  conditions  in  every  scenario  save  the  trials  with  2  RPAs.  As  the  workload,  or  number  of 
sensors,  increases,  so  will  the  operator's  reliance  on  ray  augmentation.  The  results  showed  an 
increase  in  perfonnance  in  the  4  vehicle  condition  when  the  participants  had  the  rays. 
Perfonnance  was  the  same  with  or  without  rays  in  the  2  and  6  vehicle  conditions. 

Publications 

Rowe,  A.,  Venero,  P.,  &  Boyer,  J.  (2013,  June)  Incorporating  Augmented  Reality  into  a  Tablet 
Interface  for  the  Exploitation  of  Multiple  Sensor  Feeds.  Paper  presented  at  Military  Operations 
Society  Symposium,  Alexandria,  VA. 

2.3.6  Continual  Understanding  of  Elevation  Separation  (CUES)  Study 

Introduction 

The  current  state-of-the-art  for  Remotely  Piloted  Aircraft  (RPA)  to  maintain  situational 
awareness  is  the  Tactical  Situation  Display  (TSD).  The  TSD  relies  on  symbols,  icons,  and 
dynamic  glyphs  overlaid  on  different  types  of  maps,  satellite  imagery  and  to  represent  air  traffic 
in  the  area  of  interest.  This  set-up  has  many  limitations,  one  of  which  being  that  it  is  a  2- 
dimensional  representation  of  3-dimensional  infonnation.  This  is  a  particularly  vexing  issue  as  it 
limits  operators'  ability  to  use  the  TSD  for  deconfliction  of  airspace.  This  is  mainly  because 
RPA  operators  use  altitude  as  their  main  method  of  deconfliction.  Limited  camera  views  and 
numeric  representations  of  altitude  make  it  hard  to  see  other  aircraft  and  delineate  between 
whether  or  not  they  are  roughly  at  the  same  Mean  Sea  Level  (MSL).  The  7 1 1th  Human 
Perfonnance  Wing  Supervisory  Control  Interfaces  Branch  (711  HPW/RHCI)  is  researching  the 
use  of  preattentive  cues  to  maintain  operator  awareness  of  aircraft  relative  to  the  one  under 
control.  These  cues  would  be  simple,  yet  salient,  to  allow  for  quick  reference  for  efficient  and 
reliable  deconfliction.  RHCI  will  test  various  methods  of  representing  altitudinal  differences 
with  respect  to  one's  aircraft  by  changing  the  opacity,  size,  texture,  and  enclosure  (shape  around 
the  aircraft  marker)  to  investigate  their  value  of  portraying  this  critical  information.  In  ths  way, 
RPA  operators  would  need  only  to  look  at  the  TSD  and  note  which  aircraft  are  at  the  same  flight 
level;  a  far  simpler  task  when  compared  to  the  current  method  of  looking  at  numeric  altitudes 
and  comparing  them  to  one's  own.  The  goal  of  this  study  is  to  detennine  the  most  effective  way 
to  represent  differences  between  aircraft  altitudes  relative  to  one  under  control  to  allow  for  easier 
airspace  deconfliction. 

Methods 

The  evaluation  involved  10  participants  who  completed  a  75 -question  test.  The  test  consisted  of 
5  trials  (one  per  condition)  that  had  15  questions  each.  The  15  questions  were  divided  into  3 
sets.  The  first  set  of  questions  asked  which  aircraft  were  on  the  same  level  as  that  of  the 
participant  while  only  showing  symbols  that  represented  aircraft  being  on  or  above  his/her 

15 

Distribution  A:  Approved  for  public  release;  distribution  unlimited. 

88  ABW  Cleared  08/20/2013;  88ABW-20 13-37 16. 


altitude.  The  second  asked  which  aircraft  were  on  the  same  level  as  that  of  the  participant  while 
showing  symbols  that  represented  aircraft  being  below,  above,  or  on  his/her  altitude.  The  second 
asked  which  aircraft  were  above  the  participant's  flight  level  while  showing  symbols  that 
represented  aircraft  being  below,  above,  or  on  his/her  altitude.  The  symbols  were  arranged  in  an 
8X6  grid  (no  lines)  on  a  gray  background.  The  answer  sheet  was  a  corresponding  grid  with  grid 
lines  and  points  (pencil  and  paper).  The  participants  encountered  slides  that  would  appear  for  3 
seconds  and  then  go  blank.  They  would  then  circle  the  symbols  on  the  paper  grid  relative  to  the 
ones  on  the  screen.  Grading  assessed  the  participants'  accuracy  in  noting  the  target  symbols' 

Discussion 

The  results  of  this  evaluation  provide  for  recommendations  and  future  areas  of  study.  In  all 
cases,  opacity  performed  well.  Symbol  size  also  contributed  to  good  performance.  Most  of  the 
participants  posited  that  a  combination  of  opacity  and  size  at  any  level  might  enable  the  best 
degree  of  saliency.  Altogether  opacity  would  most  likely  be  the  best  representations  of  altitude 
separation.  This  method  worked  very  well  for  representing  higher  and  lower  altitudes  as  well  as 
being  salient  enough  for  showing  which  aircraft  are  on  the  same  level.  Further  investigation  is 
warranted  in  this  area  to  establish  better  guidelines  and  practices.  Brightness  and  texture  could 
be  implemented  better  to  see  if  they  have  any  benefits.  The  hostile  symbols  could  be  evaluated 
to  determine  why  they  were  marked  less  frequently.  This  could  be  of  importance  as  it  would  be 
impractical  to  have  operators  not  notice  dangerous  entities.  Finally  these  symbols  should  be 
analyzed  overlaid  on  a  map  or  TSD  while  moving. 

Publications 

None 


2.3.7  Strategies  for  Maintaining  Awareness  of  Related  Targets  (SMART) 

Introduction 

This  study  was  aimed  at  evaluating  methods  to  represent  multiple  moving  and  stationary  targets 
for  recall  accuracy. 

Methods 

Participants  had  to  briefly  watch  a  screen  with  multiple  homogenous  points  and  then  recall  their 
locations  and  the  direction  of  movement  for  moving  targets.  A  "no  aid"  condition  was  used  as 
the  baseline.  Test  implementations  included  bars  drawn  in  the  direction  of  movement  on  moving 
icons  and  drawing  lines  between  icons  to  create  polygons  so  that  participants  could  remember 
shapes  instead  of  points.  Participants'  perfonnance  was  measured  in  tenns  of  accurately  stating 
how  many  icons  there  were  and  marking  the  icons'  locations  on  a  grid.  Analyses  gave  each 
participant  a  score. 

Discussion 

The  condition  where  bars  were  drawn  in  the  direction  of  movement  enabled  significantly  better 
perfonnance,  while  being  rated  most  favorably  by  participants.  The  polygon  condition  was  rated 
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least  favorably  and  significantly  decreased  participants'  recall  performance.  Future 
implementations  should  likely  try  to  use  directional  bars  with  color,  texture,  or  opacity  cues. 

Publications 

None 

2.4  Multi-Modal  Interactions 


Because  of  intense  information  load,  visual  clutter,  and  awkward  input  methods,  a 
pilot  can  only  control  one  UAV  at  a  time  the  multi-modal  set  of  studies  seek 
create  an  intuitive  interface  such  that  one  pilot  can  successfully  monitor  and 
manage  multiple  UAVs 


2.4.1  Determination  of  Efficiency  for  a  Variety  of  Input  Control  Equipment  (DEVICE) 
Study 

Introduction 

The  objective  of  this  study  was  to  investigate  the  effectiveness  of  several  multi-modal  input 
devices  as  alternatives  to  the  standard  mouse  for  simple  computer  screen  movements  and 
interactions  common  to  supervisory  control  of  multiple  Remotely  Piloted  Aircraft  (RPAs)  and  to 
develop  human-system  interface  design  recommendations  to  enable  efficient  and  effective 
performance.  The  devices  were  1)  standard  mouse,  2)  Belkin  n52te,  3)  Saitek  Cyborg  Command 
Unit,  4)  Wacom  Bamboo  Fun  with  Stylus,  5)  Wacom  Bamboo  Fun  with  Touch,  and  6)  Xbox  360 
Controller.  Participants  perfonned  4  tasks  with  each  device:  neutral  point  movement,  dragging, 
tracking,  and  zooming.  Participants'  self-reported  level  of  experience  varied  among  the  devices. 

Methods 

Participants  were  12  civilian  and  military  personnel  stationed  at  Wright-Patterson  Air  Force 
Base,  OH.  The  sample  consisted  of  6  men  and  6  women.  Ten  of  the  12  participants  were 
between  18-25  years  of  age  and  2  were  between  26-35  years.  Participants  were  required  to  have 
normal  visual  acuity  (20/20)  or  corrected-to-normal  visual  acuity  in  both  eyes  and  normal  color 
vision.  Visual  acuity  and  color  vision  were  determined  by  self-report.  Participation  was 
voluntary;  no  compensation  was  offered  for  participation  in  this  study. 

The  study  began  with  a  briefing  regarding  the  study  objectives  and  completion  of  the  informed 
consent  form  and  biographical  data  collection.  During  the  course  of  the  experiment,  each 
participant  performed  4  tasks  (neutral  point  movement,  tracking,  dragging,  and  zooming)  with 
each  on  the  6  devices.  The  order  of  the  devices  varied  across  participants.  However,  the  order 
of  the  tasks  was  the  same  for  each  device.  Participants  were  allowed  to  train  with  each 
device/task  combination  until  they  indicated  they  were  comfortable  with  the  task  and  input 
device.  Following  practice,  participants  completed  several  test  trials  for  each  task/device 
combination,  then  completed  a  post-device  questionnaire  regarding  the  utility  of  the  device  for 
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performing  each  task  and  their  comments  regarding  the  strengths  and  weaknesses  of  that  device. 
This  procedure  was  followed  until  each  of  the  devices  had  been  evaluated.  On  completion  of  the 
test  trials,  participants  completed  the  post-study  questionnaire. 

Discussion 

Paired  samples  t- tests  indicated  that  subjects  were  more  familiar  with  the  standard  mouse  and 
keyboard  than  any  of  the  other  devices.  Task  perfonnance  was  measured  by  average  response 
time  across  trials  for  the  neutral  point  movement,  dragging,  and  zooming  tasks.  Task 
perfonnance  for  the  tracking  task  was  measured  by  average  root  mean  square  error  across  trials. 
Sixteen  of  the  20  comparisons  between  the  standard  mouse  and  the  other  five  devices  were 
statistically  significant.  In  all  instances  where  the  difference  was  statistically  significant,  the 
standard  mouse  outperformed  the  other  devices  (i.e.,  lower  response  time,  smaller  RMS  enor). 
Although  results  strongly  favored  the  standard  mouse,  the  researchers  note  the  need  for 
additional  studies  under  more  realistic  conditions  to  determine  the  generalizability  of  the  results. 

Publications 

Aldridge,  A.C.,  Newman,  M.R,  Carretta,  T.R,  Rowe,  A.J,  French,  G.A.,  &  Whalen,  J.P.,  (2012). 
Supervisory  Control  Infonnation  Management  Research  (SCIMR)  Studies:  Determination  of 
Efficiency  for  a  Variety  of  Input  Control  Equipment  (DEVICE),  AFRL-RH-WP-TR-20 12-0050. 
Wright-Patterson  AFB,  OH:  Air  Force  Research  Laboratory,  Human  Effectiveness  Directorate, 
Crew  Systems  Interface  Division,  Supervisory  Control  Interfaces  Branch. 

2.4.2  Study  of  Hotkey  Operation  Relative  to  Touch  Commands  in  Utility  Tasks 
(SHORTCUT) 

Introduction 

Detennine  performance  characteristics  for  gesturing  and  gaming  input  control  compared  to 
current  findings  for  accepted  input  device  efficacy  (DEVICE)  to  aid  in  development  of  input 
methods  for  a  full  control  station  and  a  Man-portable  control  system.  Independent  variables- 
Method  of  input  (mouse,  Bamboo  with  stylus,  mouse  with  gaming  keyboard,  and  multi-touch); 
Dependent  variables-  Perfonnance  measured  in  numbers  of  errors  and  time  of  completion  for  a 
relatively  complex  maze  and  side  task  set-up.  Methods:  30-40  Participants,  4  Input  methods,  2 
Mazes  (Go  forward  and  back),  Post  trial  and  post  session,  Between  groups  design,  30  Mins  per 
session. 

Methods 

Participants  included  30  Civilian  and  military  employees  located  at  Wright-Patterson  AFB,  OH. 
After  examination  of  the  data,  2  participants  were  removed  from  the  analyses  due  to  extreme 
scores  on  their  response  times.  The  sample  consisted  of  18  males  and  10  females.  They  were 
placed  in  age  groups,  with  16  participants  between  18  and  25  years  of  age,  8  participants 
between  26  and  35  years  of  age,  1  participant  between  36  and  45  years  of  age  and  2  participants 
who  were  45  years  of  age  or  older.  Both  were  in  the  Touch  device  condition.  Participants  were 
required  to  have  nonnal  vision  (20/20)  or  corrected-to-nonnal  vision  in  both  eyes  and  normal 
color  vision.  Visual  acuity  and  color  vision  were  determined  by  participant  self  report. 
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Each  session  began  with  a  briefing  regarding  the  study  objectives  and  completion  of  the 
informed  consent  document  and  demographic  questionnaire.  Following  the  introduction, 
participants  underwent  training  for  their  respective  devices.  Participants  trained  until  they  met 
the  requirements  to  proceed  to  the  trial  mazes.  During  the  course  of  the  data  collection, 
participants  completed  2  trials,  one  with  a  spiral  and  one  with  an  N-shaped  maze.  Each  trial  was 
defined  by  a  maze  task  combined  with  a  set  of  5  secondary  tasks.  The  sequence  of  events  was  the 
same  for  all  participants,  but  the  maze  order  was  counterbalanced,  meaning  that  every  other 
participant  received  the  spiral  maze  before  the  N  maze.  Following  each  trial,  participants 
completed  a  NASA  TLX  for  that  maze.  At  the  conclusion  of  both  trials,  participants  completed  a 
post-session  questionnaire.  The  entire  session  required  an  average  of  an  hour  including  the 
introductory  briefing,  informed  consent,  training,  two  mazes,  and  questionnaires. 

At  the  test  station,  each  participant  received  detailed  training  in  all  of  the  procedures  to  be 
employed  during  the  entire  experiment.  First,  participants  were  given  an  introductory  briefing  on 
the  primary  and  secondary  tasks  to  include  a  description  of  objectives,  feedback  mechanisms, 
and  perfonnance  analysis  methods.  Training  progressed  from  proving  proficiency  with  each 
individual  task  (encompassing  all  of  the  maze  actions  and  the  secondary  tasks  outlined  in  the 
"Mapping"  section)  to  small  task  sequences  (i.e.,  change  color  then  engage  shield)  and, 
ultimately,  perfonning  primary  (maze)  actions  and  secondary  tasks  simultaneously.  Participants 
were  prompted  by  on-screen  text  instructions.  They  attempted  each  task/action  until  it  was 
performed  correctly  5  times  in  a  row,  or  until  a  minimum  success  rate  of  70%  out  of  10  trials  was 
achieved.  If  this  success  rate  was  not  achieved  in  the  first  10  trials,  participants  continued 
attempting  the  task/action  until  the  success  rate  increased  to  70%  or  the  task/action  was 
perfonned  correctly  5  times  in  a  row. 

Prior  to  the  beginning  of  each  trial,  there  was  a  screen  prompting  "Click  to  begin  trial".  Clicking 
caused  the  maze  to  appear.  After  a  three  second  countdown,  (visually  represented  in  the  center  of 
the  screen)  which  allowed  the  participants  to  quickly  familiarize  themselves  with  the  maze's 
layout,  the  participants  were  able  to  interact  with  the  maze  and  the  timers  began. 

The  task  set  involved  successfully  completing  a  maze  that  had  obstacles  to  overcome  and 
objectives  which  participants  had  to  accomplish.  The  maze  was  on  the  left  side  of  the  monitor 
and  additional,  secondary  tasks  were  performed  on  the  right.  Maze  completion  was  dependent 
upon  successfully  accomplishing  all  prescribed  tasks  while  still  reaching  the  end. 

The  primary  task  was  a  maze  on  the  left  side  of  the  screen.  Completion  of  the  maze  involved 
overcoming  obstacles  with  the  use  of  assigned  gestures,  hotkeys,  or  menus,  as  well  as 
completing  secondary  tasks  on  the  right  side  of  the  screen  with  the  main  device.  Secondary  tasks 
did  not  have  to  be  performed  immediately  and  could  have  been  stacked  in  a  queue.  Bimanual 
coordination  was  encouraged  by  putting  a  limit  on  the  number  of  secondary  tasks  available  for 
completion;  if  the  queue  reached  five  tasks,  participants  were  unable  to  make  progress  in  maze 
completion  until  a  secondary  task  was  perfonned,  preventing  them  from  focusing  solely  on  the 
primary  task. 

Designs  for  primary  and  secondary  tasks  were  derived  from  integrating  Fitts'  law  and  steer  point 
analysis  in  conjunction  with  a  test  created  by  Armbruster  et  al.  (2007).  Other  design 
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considerations  were  derived  from  contemporary  and  foreseeable  multi-RPA  supervisory  control 
and  sensor  management  environments. 

Discussion 

There  were  no  significant  differences  between  the  mouse  with  Belkin  n52te  and  the  standard 
mouse,  but  the  mouse  with  Belkin  n52te  trended  better  than  the  standard  mouse  alone  in  maze 
completion  time,  reactions  to  stimuli,  and  error  rates.  The  multi-touch  functionality  was 
significantly  worse  than  the  mouse  with  Belkinn52te  and  provided  many  lessons  learned.  The 
researchers  identified  the  need  for  more  flexible  gesture  recognition,  should  complex  gestures  be 
used.  The  optimal  method  for  utilizing  multi-touch  in  its  current  state,  however,  is  to  use  simple, 
single-finger  inputs  to  ensure  higher  accuracy  and  lower  fatigue  and  frustration. 

Publications 

Aldridge,  A.C.,  Newman,  M.R,  Carretta,  T.R,  Rowe,  A.J,  French,  G.A.,  &  Whalen,  J.P.,  (2012). 
Supervisory  Control  Infonnation  Management  Research  (SCIMR)  Studies:  Study  of  Hotkey 
Operations  Relative  to  Touch  Commands  in  Utility  Tasks(SHORTCUT),  AFRL-RH-WP-TR- 
2012-0168.  Wright-Patterson  AFB,  OH:  Air  Force  Research  Laboratory,  Human  Effectiveness 
Directorate,  Crew  Systems  Interface  Division,  Supervisory  Control  Interfaces  Branch. 

2.4.3  Surveilling  an  Urban  Populous  Area  with  a  Robust  Monitoring  Asset  Network 
(SUPARMAN) 

Introduction 

The  objective  of  this  research  is  to  identify  the  best  input  techniques  for  managing  an  urban 
electro-optical  sensor  network  with  a  single  user.  The  first  study  in  what  should  be  a  series  will 
build  off  of  the  VOICE  research  and  evaluate  the  performance  gains  using  speech  to  augment 
multi-touch.  In  this  case,  as  with  VOICE,  vocal  input  used  concurrently  with  touch  will  be 
assessed  in  comparison  to  touch  alone.  The  participants  will  have  at  their  command  20  fixed 
urban  security  cameras  as  well  as  a  remotely  piloted  aircraft  sensor  gimble.  The  task  set  will 
involve  identifying  a  target  in  a  densely  populated  simulated  city,  tracking  the  target,  and 
capturing  still  images  of  the  target  as  he/she  moves  throughout  the  city. 

Methods 

This  study  will  require  20  participants.  The  subject  pool  will  be  limited  to  members  of  the  U.S. 
Air  Force,  U.S.  DoD  civilian  employees,  U.S.  DoD  contractors,  other  Anned  Services  units,  and 
paid  volunteers.  Participants  are  required  to  have  nonnal  vision  (20/20)  or  corrected-to-normal 
vision  in  both  eyes,  nonnal  peripheral  vision,  and  normal  color  vision.  Visual  acuity  and  color 
vision  will  be  detennined  by  subject  self  report  (in  response  to  the  invitation).  This  study  will 
use  a  within  groups  design  in  which  the  two  scenarios  and  configurations  will  be 
counterbalanced  to  prevent  expectancy  errors.  Screening  for  gender  or  a  specific  male/female 
ratio  will  be  administered  for  a  50/50  split.  Participants  must  be  18  years  old  or  older. 

Participants  will  first  be  asked  to  identify  the  target  in  a  simulated  urban  environment.  The 
simulated  environment  will  be  replete  with  people  and  automobiles.  The  basic  description  of  the 
target  will  match  the  visual  representations  of  5  simulated  people  among  the  city's  population. 
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The  target  will  be  identified  when  he  or  she  stops  and  looks  at  his/her  watch.  Until  that  action 
occurs,  the  participants  will  have  to  find  and  then  monitor  all  five  potential  targets.  Participants 
will  have  around  6  minutes  to  identify  and  track  potential  targets  before  the  actual  target 
performs  the  specified  action.  When  the  participants  identify  the  target,  they  will  then  select  and 
manipulate  the  appropriate  sensors  to  maintain  coverage  while  capturing  images  when  required. 

After  8  minutes  of  tracking  the  walking  target,  he/she  will  begin  to  run  on  a  generated  path.  The 
participants  will  have  to  then  maintain  coverage  on  the  faster  moving  target.  After  a  few  more 
minutes,  the  target  will  enter  a  vehicle  and  the  participants  will  have  to  track  that  vehicle  as  it 
moves  considerably  faster.  To  aid  the  participant,  radio  calls  will  come  from  simulated  human 
agents  on  the  ground  that  will  announce  possible  sightings.  These  radio  calls  will  be  correct 
80%  of  the  time. 

Along  with  tracking,  participants  will  have  the  task  of  capturing  still  images  (via  the  main  sensor 
window)  of  three  important  events.  These  events  are:  the  target  checking  his/her  watch,  picking 
up  an  object,  entering  a  vehicle,  and  exiting  the  area  of  regard.  Participants  will  be  given 
direction  to  attempt  to  catch  these  actions  in  progress  for  evidence. 

Discussion: 

The  data  collected  showed  that  overall  the  interface  layout  is  user-friendly.  Touch  was  found  to 
be  more  intuitive  than  speech.  Voice  commands  would  make  the  task  easier  to  perfonn  if  more 
exposure  was  given  to  the  commands  prior  to  task  execution.  Voice  commands  often  had  to  be 
repeated  (speech  software  not  always  accurate),  which  wasted  time  and  increased  workload.  A 
speech-driven  HMI  would  be  distracting  to  others  in  the  actual  work  environment  and  would 
complicate  parsing.  A  physical  joystick  would  be  more  intuitive  for  sensor  window  functions 
(pan/tilt/zoom).  Providing  both  an  exo  and  ego  viewpoint,  the  "touched  point"  feature,  the 
Sensor  Window  Dock,  field  agent  calls  (linked  to  map)  all  helped  maintain  SA  and  increased 
time  on  target. 

Publications 

Roll,  J.,  Lampke,  S.  &  Adkins,  D.  (2013,  June)  Surveillance  of  an  Urban  Populous  Area  with  a 
Robust  Monitoring  Network  of  Sensor  Assets.  Paper  presented  at  Military  Operations  Society 
Symposium,  Alexandria,  VA. 

3.0  CONCLUSIONS 

Unmanned  aerial  systems  (UAS)  are  becoming  an  increasingly  critical  aspect  of  military 
operations.  To  meet  this  demand,  the  USAF  is  seeking  ever  more  capable  UASs,  to  include  the 
ability  of  a  single  operator  to  simultaneously  control  multiple  platforms,  increased  connectivity 
to  net-centric  infonnation  sources,  and  the  ability  to  accomplish  more  complex,  dynamic 
missions.  These  capabilities  include: 

1)  close  collaboration  with  manned  assets 

2)  find,  fix,  track,  and  target  difficult  targets  in  complex  urban  and  difficult 
terrains 

3)  destroy  or  neutralize  difficult  targets 
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4)  precisely  deliver  select  effects  to  maintain  controllable  collateral  damage 

5)  persistence  in  multiple  areas  of  interest 

To  fulfill  these  missions,  the  USAF  is  exploring  multi-vehicle  UAS  concepts  to  carry  out  tactical 
intelligence,  surveillance,  reconnaissance  and  combat  missions.  In  many  of  these  concepts  there 
is  an  emphasis  on  managing  UAS  systems  and  conducting  missions  with  minimal  crew.  UASs 
have  evolved  from  being  primarily  remotely  controlled  systems  to  being  pre-programmed  or 
semi-autonomous,  changing  the  role  of  the  crew  from  flying  to  supervising.  This  change  has 
opened  the  door  to  increasing  the  vehicle  to  operator  ratio.  While  progress  has  been  made  in 
developing  more  capability  with  multi-vehicle  systems  (e.g.,  more  simultaneous  vehicle  orbits 
managed  from  one  control  station),  further  research  is  needed  to  increase  mission  effectiveness 
on  per  vehicle  and  per  operator  basis.  To  increase  mission  effectiveness,  crew  perfonnance  and 
capability  enhancements  are  needed.  Technology  development  and  advanced  designs  are 
required  to  facilitate  more  timely  and  effective  operator  situation  assessment  and  decision¬ 
making. 
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